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Executive  Summary 

This  report  compares  the  contents  of  85  different  Internet  blacklists,  also  known  as 
threat  intelligence  feeds  or  threat  data  feeds,  to  discover  patterns  in  shared  entries. 
It  is  an  update  to  a  2013  report  that  compared  25  such  Internet  blacklists  [1], 
The  methods  and  motivations  of  this  report  are  similar  to  those  employed  in  the 
earlier  report.  However,  this  update  provides  an  expanded  scope  by  increasing  the 
number  of  lists  and  the  duration  of  the  investigation  by  another  year.  This  report 
does  not  contain  the  same  depth  of  detail  as  the  2013  report,  especially  where 
details  have  not  changed.  See  the  prior  report  at  http://url.sei.cmu.edu/BL-13. 

Lists  are  compared  directly  and  indirectly,  based  on  data  type.  Direct  intersec¬ 
tion  comparison  is  straightforward;  the  list  contents  are  compared  temporally  to 
determine  if  any  list  consistently  published  shared  indicators  before  another  list. 
Indirect  comparison  analyzes,  for  example,  whether  the  existing  intersection  is 
random  or  has  a  pattern. 

These  multiple  methods  indicate  a  range  for  how  often  a  list  provides  an  in¬ 
dicator  with  unique  information  and  value  to  computer  network  defense  (CND). 
Domain-name-based  indicators  are  unique  to  one  list  between  96. 16%  and  97.37% 
of  the  time.  IP-address-based  indicators  are  unique  to  one  list  between  82.46%  and 
95.24%  of  the  time. 

These  2014  results  support  our  2013  results  and  conclusions,  and  are  gen¬ 
erally  consistent.  Namely,  there  is  surprisingly  little  overlap  between  any  two 
blacklists.  Though  there  are  exceptions  to  this  pattern,  the  intersection  between 
the  lists  remains  low,  even  after  expanding  each  list  to  a  larger  neighborhood  of 
related  indicators.  Few  lists  consistently  provide  content  before  certain  other  lists, 
but  more  often  there  is  no  intersection  at  all.  When  there  is  an  intersection,  many 
times  there  is  no  pattern  to  which  list  came  first. 

These  results  suggest  that  each  blacklist  describes  a  distinct  sort  of  malicious 
activity.  The  lists  do  not  appear  to  converge  on  one  version  of  all  the  malicious 
indicators  for  the  Internet.  Network  defenders  should  be  advised,  therefore,  to  ob¬ 
tain  and  evaluate  as  many  lists  as  practical,  since  it  does  not  appear  that  any  new 
list  can  be  rejected  out-of-hand  as  redundant.  The  results  also  indicate  that  there 
is  no  global  ground  truth  to  be  acquired,  no  matter  how  many  lists  are  merged. 
Therefore,  the  study  supports  the  assertion  that  blacklisting  is  not  a  sufficient 
defense;  an  organization  needs  other  defensive  measures  to  add  depth,  such  as 
gray  listing,  behavior  analysis,  criminal  penalties,  speed  bumps,  and  organization- 
specific  white  lists. 

This  analysis  provides  a  collective  view  of  the  whole  ecosystem  of  block¬ 
ing  network  touch  points  and  blacklists.  Many  practitioners  lament  the  fatigue 
of  playing  “whack-a-mole”  against  very  resilient  adversary  resources.  This  tacit 
knowledge  must  be  formalized  before  a  better  collective  strategy  can  be  enacted. 
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The  blacklist  ecosystem  supports  this  tacit  knowledge  and  formalizes  a  part  of  it: 
since  lists  are  largely  distinct,  “whack-a-mole”  is  inevitable  and  impossible.  With¬ 
out  convergence,  practitioners  are  left  to  do  the  best  they  can  with  the  extensive 
but  fragmentary  blacklist  data  that  is  available. 

Blacklist  ecosystem  analysis  is  one  aspect  of  a  larger  body  of  work  to  quantify 
strategic  cybersecurity  issues.  The  blacklist  ecosystem  is  intimately  related  to 
the  low  cost  of  domains  and  infrastructure  to  adversaries  [2],  the  poor  state  of 
repair  of  consumer  devices  connected  to  the  Internet  that  permits  abuse  [3],  the 
challenges  of  modeling  the  interaction  between  the  user  and  the  adversary  [4],  and 
the  challenges  of  designing  effective  and  instructive  observations  in  information 
security  [5], 

1  Motivation 

The  2013  blacklist  ecosystem  report  started  quite  a  few  conversations.  It  was  clear 
the  community  wanted  an  expanded  analysis  to  verify  and  solidify  the  results.  Be¬ 
yond  this  aim,  the  motivation  remains  largely  the  same  and  this  section  is  adapted 
from  the  prior  work  [1].  Almost  every  organization  engaged  in  cybersecurity  uses 
blacklists,  but  effectiveness  is  impossible  to  quantify.  Blacklist  ecosystem  analy¬ 
sis  cannot  evaluate  individual  list  effectiveness;  however,  practitioners  can  learn 
plenty  from  quantifiable  properties  of  the  ecosystem  of  blacklists  and  the  interre¬ 
lationships  among  lists. 

Although  there  are  quite  a  few  organizations  that  provide  blacklists,  there  is 
little  information  about  how  various  lists  are  produced.  This  secrecy  is  justified 
because  most  providers  are  engaged  in  a  battle  of  wits  with  adversaries.  Disclo¬ 
sure  of  the  precise  procedure  of  generating  the  lists  risks  of  the  quality  of  the 
lists.  However,  this  secrecy  does  not  benefit  the  operational  analyst  who  must 
decide  which  lists  to  apply  on  which  network  access  control  points  and  is  often 
left  making  semi-educated  guesses  about  the  providence  and  usefulness  of  a  list 
in  a  particular  situation.  We  previously  identified  this  interaction  between  the 
(list)  architect,  user,  and  adversary  as  requiring  further  study  [4],  and  the  blacklist 
ecosystem  helps  to  inform  that  broader  effort. 

From  an  operational  point  of  view,  the  question  is  quite  practical.  Network 
defenders  need  to  know  which  lists  they  should  use  to  defend  their  networks. 
Evaluating  individual  lists  is  not  generally  possible  because  there  is  no  global 
ground  truth  about  maliciousness.  Ecosystem-wide  views  of  blacklist  interaction 
is  informative  for  the  practitioner.  If  no  lists  overlap,  and  few  mimic  one  another, 
then  the  strategy  would  appear  to  be  to  acquire  all  lists,  as  they  all  contain  unique 
value. 

Blacklist  interrelation  affects  the  information  security  evaluation  and  baseline 
creation  as  well.  Academic  and  industry  papers  often  rate  performance  of  a  par¬ 
ticular  task  according  to  its  agreement  with  some  blacklist  or  lists.  If  all  lists  were 
equal  or  generation  methods  open,  this  method  would  be  acceptable.  However, 
because  each  list  is  different  and  largely  non-overlapping,  the  ability  to  alter  re¬ 
sults  by  the  choice  of  list  leaves  the  evaluation  process  open  to  manipulation,  as 
an  author  can  choose  the  list  that  offers  the  best  agreement. 


2  Method 

List  acquisition  and  comparison  methods  are  largely  the  same  as  in  the  prior  re¬ 
port  [1].  Basic  results  include  reverse  counts,  list  size  measurements,  and  pairwise 
intersections.  Notable  results  reported  here  include  which  lists  appear  to  be  fol¬ 
lowing  other  lists.  Methods  for  these  processes  are  described  in  this  section. 

List  acquisition  includes  all  the  unique  indicators  in  a  list  from  March  16, 
2013  to  June  30,  2014  (essentially  2Q2013-2Q2014),  or  15  months.  List  acqui- 
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sition  for  the  2013  blacklist  report  ended  on  March  31,  2013;  therefore,  this  new 
data  analysis  creates  a  consecutive  date  range  of  30  months  of  blacklist  content 
analysis. 

List  acquisition  has  potential  inconsistencies.  For  example,  our  list  acquisition 
was  not  constant.  Lists  were  acquired  at  certain  time  points,  and  each  list  could 
not  be  acquired  at  exactly  the  same  time.  This  asynchrony  makes  determining 
who  listed  what  first  difficult;  therefore,  we  worked  in  units  of  days  when  deter¬ 
mining  “at  the  same  time”  and  treated  anything  on  the  same  day  as  equivalent.  In 
some  cases,  list  providers  limited  downloads  to  once  per  day;  whereas,  others  en¬ 
couraged  two  or  three  daily  downloads.  If  an  indicator  was  listed  only  in  between 
downloads,  it  would  not  be  observed.  We  judged  that  these  inconsistencies  are  not 
relevant  to  the  granularity  at  which  we  are  comparing  the  lists. 

Comparison  across  such  large  time  windows  has  certain  potential  pitfalls,  es¬ 
pecially  for  IP  addresses  based  on  how  they  are  used  on  the  Internet.  Over  time,  IP 
addresses  are  reassigned  and  reused  due  to  features  such  as  NAT,  DHCP,  BGP,  and 
IP  address  stewardship  or  assignment  changes  from  the  regional  Internet  registries 
(RIRs). 

We  expect  that  these  mechanisms  have  a  real  impact  on  measurement  over 
more  than  one  year.  All  of  these  technical  features  have  the  effect  of  apparently 
and  erroneously  increasing  the  intersection  between  lists.  The  increase  in  inter¬ 
section  is  because  the  same  identifier  is  used  by  multiple  machines,  and  the  lists 
may  be  detecting  activity  from  a  machine  for  each  identifier  it  has.  Alternatively, 
if  an  identifier  is  shared  by  multiple  machines,  two  lists  may  detect  distinct  behav¬ 
ior  from  distinct  machines,  but  appear  to  intersect  because  those  machines  share 
an  identifier.  These  impacts  generally  serve  to  make  the  reverse  count  analysis  an 
upper  bound  for  how  much  intersection  there  is  between  lists.  We  account  for  the 
effect  of  this  overestimation  analytically  in  Section  4. 

2.1  Reverse  Counts 

The  method  used  for  counting  how  many  indicators  are  unique  to  one  list,  two 
lists,  three  lists,  etc.,  is  straightforward.  Each  comparable  indicator  (i.e.,  all  the 
IP  addresses)  is  tagged  with  how  many  lists  contained  it.  The  number  of  lists  per 
indicator  is  counted;  call  it  n.  The  reported  result  is  the  number  of  indicators  on  n 
lists  for  n  =  1  up  to  the  maximum  n  observed. 

2.2  List  Counts 

List  counts  are  the  total  number  of  unique  indicators  observed  on  the  list  at  any 
time  during  the  observation  period.  Each  list  is  given  an  anonymized  numeric 
identifier  and  labeled  either  LI  for  a  list  of  IP  addresses  or  LD  for  a  list  of  do¬ 
main  names.  This  naming  convention  is  used  wherever  lists  must  be  referred  to 
individually.  Each  list’s  identifier  is  the  same  throughout  the  report. 

2.3  Pairwise  Intersection  Counts 

Each  possible  pairing  of  lists  is  generated  and  the  cardinality  of  the  intersection 
between  the  two  sets  is  reported.  With  18  domain-name-based  lists,  there  are  (*28) 
or  162  pairings.  With  67  IP-address-based  lists  there,  are  (67)  or  2244  pairings. 

2.4  Following 

For  lists  that  had  intersections  of  greater  than  1000  elements,  we  performed  a 
one-sample  t-test  to  determine  whether  it  seemed  that  one  list  was  consistently 
publishing  elements  before  another  list.  We  calculated  this  determination  on  the 
granularity  of  one  calendar  day,  not  per  second.  Due  to  our  collection  delays,  a 
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coarser  granularity  is  necessary  to  reduce  the  effects  caused  by  collection  idiosyn¬ 
crasies  and  to  isolate  genuine  effects. 

If  neither  list  followed  the  other,  we  would  expect  any  intersection  to  be  essen¬ 
tially  random,  with  as  many  elements  discovered  first  by  list  1  as  by  list  2.  If  each 
list  were  to  find  as  many  earlier  as  later,  the  average  difference  between  shared 
element  discovery  times  would  be  0.  Therefore,  we  hypothesize  the  average  mean 
of  the  deltas  to  be  0,  and  we  can  test  this  result  to  calculate  the  probability  it  is 
true,  based  on  the  sample.  If  we  can  reject  this  null  hypothesis  of  a  0  mean,  we 
have  reason  to  believe  that  one  list  is  following  the  other. 

The  t-test  is  calculated  as  follows.  For  each  shared  element  between  the  lists, 
a  time  delta  t\  is  calculated  as  =  t\  — tj,  where  1\  and  fi  are  the  times  list  1  and 
list  2  published  the  element,  respectively.  Over  all  shared  elements,  this  difference 
creates  a  list  of  deltas  tlA  through  t\,  where  n  is  the  number  of  shared  elements; 
call  this  set  T&.  The  t-test  is  set  to  test  that  the  mean  of  T\  is  0,  so  we  set  /To  =  0. 
We  calculate  x  as  the  mean  of  T&  and  .v  as  the  standard  deviation  of  Ta-  The  value 
of  the  t-test  for  each  list  pairing  is  calculated  as  in  Equation  1: 


x~ji  o 

s/ v« 


(1) 


The  p-value  is  calculated  by  the  standard  single-value,  two-tailed  t-test  based 
on  the  degrees  of  freedom  n—  1 .  The  result  is  the  probability  p  that  the  experi¬ 
mental  results  are  observed  by  chance  even  though  the  null  hypothesis  is  true  if 
we  repeated  the  same  experiment.  There  is  only  one  blacklist  ecosystem,  so  we 
must  test  certainty  this  way  rather  than  repeating  the  measurement.  We  discuss 
what  it  means  for  the  null  hypothesis  to  be  false  (x  /  0)  in  Section  4. 

A  summary  goal  is  to  report  on  the  number  of  indicators  involved  in  a  nonzero- 
mean  relationship  between  two  lists.  We  are  unaware  of  a  precedent  for  what 
should  be  considered  a  reasonable  p-value  in  science  of  security  work  such  as 
this.  Initially  we  tested  a  p-value  of  0.01.  At  this  value,  we  failed  to  reject  the 
null  hypothesis  for  2  of  21  domain-name-based  intersections  and  54  of  859  IP- 
address-based  intersections;  i.e.  most  results  were  significant.  However,  after 
inspecting  the  results  we  feared  this  choice  of  P  risked  a  high  type  I  error  (a). 
When  summarizing  the  results,  we  set  a  more  aggressive  p-value  for  certainty  that 
the  mean  was  nonzero:  2.2  x  1 0  1 6 .  This  p-value  is  the  lowest  real  value  that  R 
reports  for  the  test,  so  the  threshold  is  as  aggressive  as  possible.  However,  since  we 
cannot  re-run  the  test  this  year  (there  is  only  one  blacklist  ecosystem),  the  results 
should  be  considered  as  exploratory  analysis  rather  than  a  formal  hypothesis  test. 

We  only  considered  pairwise  intersections  with  more  than  1,000  elements  to 
ensure  that  the  sample  was  robust  and  to  help  control  for  anomalous  small  inter¬ 
sections.  The  indicators  from  any  pairwise  intersection  that  pass  this  test  have 
some  non-random  relationship.  Each  pairwise  intersection  provides  indicators; 
we  report  on  the  total  unique  indicators  involved  in  any  such  potential  follow¬ 
ing  relationship  by  reporting  the  cardinality  of  the  union  of  the  set  of  indicators 
involved  in  any  pairwise  intersection  passing  this  test. 


3  Results 

The  results  presented  in  this  section  are  more  concise  than  the  results  from  the 
2013  report.  This  conciseness  is  partly  because  the  results  are  largely  compatible 
with  prior  results  and  so  do  not  need  to  be  repeated.  Furthermore,  since  the  num¬ 
ber  of  lists  analyzed  increased  by  over  three-fold  to  85,  we  cannot  report  as  many 
detailed  results  and  need  to  focus  more  on  summarizing  the  results  in  meaningful 
ways. 

For  example,  we  checked  to  see  if  any  of  the  blacklisted  IP  addresses  were 
known  sinkhole  IP  addresses.  This  information  would  essentially  invalidate  the 
indicator  as  an  indicator  of  malicious  activity,  since  sinkholes  are  operated  by 
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network  defenders  who  clean  up  and  collect  intelligence  on  threats.  Only  one  list 
out  of  67,  LI_3,  contained  any  sinkhole  IP  addresses  and  that  list  contained  only 
10. 

All  the  reported  results  are  meant  to  inform  the  extent  of  uniqueness  of  black 
lists.  The  reverse  counts  indicate  how  frequently  indicators  were  on  multiple  lists. 
List  counts  give  a  sense  of  the  variety  of  lists  involved.  Pairwise  intersections 
provide  a  more  detailed  look  at  how  large  the  intersection  is  between  each  pair  of 
lists,  demonstrating  that  a  few  lists  overlap  quite  a  lot.  The  analysis  of  “following” 
attempts  to  quantify  these  pairwise  interactions  to  determine  whether  there  is  a 
reliable  cause  or  predictable  ordering  of  which  list  produces  an  indicator  first,  or 
if  the  two  lists  just  happen  to  be  listing  the  same  indicators  essentially  randomly. 

3.1  Reverse  Counts 

For  domain  names,  30,784,571  total  unique  indicators  were  observed  during  the 
15-month  observation  period.  There  were  29,602,108  indicators  observed  on  ex¬ 
actly  one  list.  There  were  1,182,463  domain  names  observed  on  multiple  lists,  or 
3.84%  of  all  observed  domain-name  indicators.  Of  the  indicators  that  appeared 
on  multiple  lists,  780,162  indicators  appeared  on  exactly  two  lists,  or  66%  of  the 
indicators  that  appeared  more  than  once.  Table  1  displays  the  complete  results  for 
how  often  domain-name  indicators  appeared  on  multiple  lists. 


#  Lists 

Count 

Ratio 

1 

29602108 

0.96158910 

2 

780162 

0.02534263 

3 

163768 

0.00531981 

4 

94065 

0.00305559 

5 

67677 

0.00219841 

6 

41195 

0.00133817 

7 

21702 

0.00070496 

8 

9401 

0.00030538 

9 

3420 

0.00011109 

10 

920 

0.00002989 

11 

138 

0.00000448 

12 

14 

0.00000045 

13 

1 

0.00000003 

Table  1:  Reverse  count  of  the  number  of  times  each  domain  is  on  domain-based  blacklists. 
(Out  of  30784571  total  domains  on  18  lists,  over  96%  were  unique  to  one  list  over  15 
months.) 

For  IP  addresses,  121,921,509  total  unique  IP  address  indicators  were  ob¬ 
served  during  the  15-month  observation  period.  There  were  100,532,890  indi¬ 
cators  observed  on  exactly  one  list.  There  were  21,388,619  IP  address  indicators 
observed  on  more  than  one  list,  or  17.54%,  with  almost  half  of  those  (10,412,833) 
occurring  on  exactly  two  lists.  Table  2  displays  the  complete  results  for  how  often 
IP-address  indicators  appeared  on  multiple  lists. 


3.2  List  Counts 

The  size  of  the  lists  surveyed  varies  widely.  Some  lists  have  over  ten  million  indi¬ 
cators,  some  have  less  than  a  thousand,  and  most  are  in  between.  The  list  names 
are  anonymized  and  given  a  random  identifier;  LD  indicates  a  list  of  domains, 
whereas  LI  indicates  a  list  of  IP  addresses.  Results  are  based  on  the  number  of 
unique  identifiers  observed  over  the  15-month  observation  period,  regardless  of 
how  long  the  identifier  was  on  the  list.  Table  3  provides  the  sizes  of  all  lists 
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#  Lists 

Count 

Ratio 

1 

100532890 

0.82457058 

2 

10412833 

0.08540604 

3 

3699338 

0.03034196 

4 

2153492 

0.01766294 

5 

1407801 

0.01154678 

6 

986683 

0.00809277 

7 

716422 

0.00587609 

8 

531285 

0.00435760 

9 

392986 

0.00322327 

10 

288769 

0.00236848 

11 

211412 

0.00173400 

12 

153286 

0.00125725 

13 

111568 

0.00091508 

14 

81692 

0.00067004 

15 

60492 

0.00049616 

16 

45576 

0.00037381 

17 

33681 

0.00027625 

18 

25552 

0.00020958 

19 

19157 

0.00015713 

#  Lists 

Count 

Ratio 

20 

14568 

0.00011949 

21 

11246 

0.00009224 

22 

8514 

0.00006983 

23 

6662 

0.00005464 

24 

5309 

0.00004354 

25 

3990 

0.00003273 

26 

2798 

0.00002295 

27 

1674 

0.00001373 

28 

995 

0.00000816 

29 

429 

0.00000352 

30 

208 

0.00000171 

31 

102 

0.00000084 

32 

59 

0.00000048 

33 

18 

0.00000015 

34 

8 

0.00000007 

35 

7 

0.00000006 

36 

4 

0.00000003 

37 

2 

0.00000002 

38 

1 

0.00000001 

Table  2:  Reverse  count  of  the  number  of  times  each  IP  address  is  on  IP-address-based 
blacklists.  (Out  of  121921509  total  IP  addresses  on  67  lists,  over  82%  were  unique  to  one 
list  over  15  months.) 


of  domain-name-based  indicators.  Table  4  provides  the  sizes  of  all  lists  of  IP- 
address-based  indicators. 


List 

Unique  Entries 

LD_1 

411871 

LD_2 

24103937 

LD_3 

55110 

LD_4 

83884 

LD_5 

73351 

LD_6 

47790 

LD_7 

67025 

LD_8 

3498 

LD_9 

499358 

List 

Unique  Entries 

LD_10 

251044 

LD_1 1 

2802602 

LD_12 

1442233 

LD_13 

173 

LD_14 

2738773 

LD_15 

61424 

LD_16 

2559 

LD_17 

178632 

LD_18 

61088 

Table  3:  Unique  entries  over  the  observation  period  for  each  list  of  domains. 


3.3  Pairwise  Intersections 

The  results  for  the  pairwise  intersections  of  all  lists  is  quite  long.  Table  5  and 
Table  6  present  these  results  in  the  Appendix.  The  lists  are  anonymized  following 
the  same  pattern  as  described  in  Section  3.2. 

3.4  Following 

The  dataset  is  not  clean  enough  to  conclude  with  certainty  that  one  list  follows 
another.  However,  where  two  lists  intersect,  we  can  tell  whether  or  not  the  lists 
appear  independent  of  one  another.  Our  “following”  test  fails  to  reject  the  null 
hypothesis  if  the  temporal  intersection  features  between  lists  appears  dependent 
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List 

Unique  Entries 

LI_1 

22250 

LI_2 

62884574 

LI_3 

3738277 

LI_4 

863 

LI_5 

72644 

LI_6 

16024 

LI_7 

18878208 

LI_8 

10378 

LI_9 

615914 

LI_10 

5858 

LI_11 

51309 

LI_12 

3024492 

LI_13 

551965 

LI_14 

134890 

LI_15 

2355 

LI_16 

3462 

LI_17 

6795 

LI_18 

60403 

LI_19 

4432 

LI_20 

10975 

LI_21 

5738359 

LI_22 

160605 

LI_23 

1 142022 

LI_24 

2702 

LI_25 

119353 

LI_26 

40051 

LI_27 

1448865 

LI_28 

597228 

LI_29 

58707 

LI_30 

3794 

LI_31 

1746662 

LI_32 

10756 

LI_33 

3705188 

LI_34 

44729 

List 

Unique  Entries 

LI_35 

32612 

LI_36 

8565 

LI_37 

13463 

LI_38 

32294176 

LI_39 

2093 

LI_40 

359251 

LI_41 

351799 

LI_42 

3552898 

LI_43 

522814 

LI_44 

171776 

LI_45 

776793 

LI_46 

444116 

LI_47 

246350 

LI_48 

11145061 

LI_49 

9638563 

LI_50 

4309163 

LI_51 

689524 

LI_52 

703105 

LI_53 

4200727 

LI_54 

2342 

LI_55 

58097 

LI_56 

25068 

LI_57 

4201662 

LI_58 

4514 

LI_59 

1752202 

LI_60 

53189 

LI_61 

1261 

LI_62 

25418 

LI_63 

255558 

LI_64 

4418 

LI_65 

8048 

LI_66 

4027 

LI_67 

3955 

Table  4:  Unique  entries  over  the  observation  period  for  each  list  of  domains. 


on  the  lists’  interaction.  This  interaction  may  be  due  to  following  or  to  some  other 
hidden  variable  that  is  influencing  one  list  to  consistently  list  an  indicator  before 
another. 

The  total  number  of  unique  domain  names  in  a  set  that  failed  the  hypothe¬ 
sis  test  of  a  zero  mean  for  the  pairwise  intersection  is  809,394,  or  68.45%  of  the 
1,182,463  indicators  that  appeared  on  multiple  lists.  There  were  17  pairwise  inter¬ 
sections  of  domain-name-based  lists  that  contributed  to  this  total,  out  of  21  total 
pairwise  list  intersections  with  more  than  1,000  elements. 

The  total  number  of  unique  IP  addresses  in  a  set  that  failed  the  hypothesis 
test  of  a  zero  mean  for  the  pairwise  intersection  is  5,803,501,  or  27.13%  of  the 
21,388,619  indicators  that  appeared  on  multiple  lists.  There  were  648  pairwise 
intersections  of  IP-address-based  lists  that  contributed  to  this  total,  out  of  859 
total  pairwise  list  intersections  with  more  than  1,000  elements. 
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4  Conclusion 

There  are  many  common  blacklists  that  describe  indicators  of  malicious  activity 
for  the  Internet.  These  lists  do  not  intersect  to  a  large  degree.  Therefore,  it  appears 
that  these  lists  do  not  converge  on  one  set  of  malicious  indicators.  For  compre¬ 
hensive  detection,  it  is  best  to  consider  all  the  lists  together  than  to  rely  on  an 
intersection. 

Although  IP  address  movement  and  reassignment  can  be  estimated  for  the 
Internet  as  a  whole,  we  cannot  reliably  estimate  the  probability  that  any  single 
IP  address  was  reassigned.  These  mechanisms  inflate  the  amount  of  intersection 
by  some  factor.  Such  an  effect  does  not  compromise  our  conclusions  because 
the  relevant  aspect  of  our  conclusion  is  how  little  intersection  there  is  between 
the  lists.  The  unknown  intersection  inflation  factor  reduces  our  confidence  in  the 
measurement  to  a  sure  upper  limit  on  the  intersection.  This  limit  is  sufficient  to 
demonstrate  that  the  intersection  is  relatively  small.  This  relatively  small  inter¬ 
section  is  consistent  with  recent  results  about  small  overlap  among  open-source 
cyber-intelligence  indicators  as  well  [6], 

Competition  among  list  vendors  appears  to  also  inflate  the  amount  of  genuine 
intersection  among  lists.  If  an  indicator  was  involved  in  a  pairwise  intersection 
that  passed  our  “following”  test,  there  is  some  interaction  between  the  two  lists 
in  the  pair.  This  interaction  may  be  one  list  explicitly  copying  the  other  list,  and 
therefore  always  appearing  later.  In  general,  the  test  indicates  only  that  there  is 
some  other  factor  we  have  not  accounted  for  making  the  lists  related  in  a  pre¬ 
dictable  way.  This  result  can  be  used  to  estimate  how  many  indicators  an  orga¬ 
nization  would  need  to  acquire  for  “complete”  coverage;  random  factors  such  as 
the  inflationary  Internet  features,  like  DHCP  and  NAT  listed  above,  should  not 
usually  cause  this  “following”  behavior. 

The  indicators  from  “following”  relationships  appear  to  be  duplicate;  all  others 
may  genuinely  be  useful  at  the  time  of  release.  If  this  reasoning  holds,  which 
requires  some  further  research  to  be  sure,  then  only  809,394  domains  (2.63%  of 
total)  and  5,803,501  IP  addresses  (4.76%  of  total)  are  actually  duplicative.  All 
others  would  be  necessary  to  acquire  as  complete  a  view  as  possible. 

The  naive  reverse  counts  do  not  account  for  any  inflationary  Internet  features. 
Our  “following”  test  is  likely  too  strict  and  undercounts  the  duplicative  results 
from  lists  because  of  the  low  p-value  used  in  the  test  and  the  artificial  limit  of 
testing  only  intersections  with  at  least  1,000  indicators.  Therefore,  we  believe  the 
genuine  result  is  somewhere  in  the  range  created  by  the  two  methods. 

The  range  of  unique  value  to  CND  from  an  indicator  on  domain-name-based 
lists  is  narrower  than  that  for  IP-address-based  lists,  but  both  ranges  indicate 
highly  unique  indicators.  Domain-name-based  indicators  from  an  average  list  do 
not  provide  unique  value  to  CND  between  2.63%  and  3.84%  of  the  time.  That 
is,  between  96.16%  and  97.37%  of  domain-name-based  indicators  are  uniquely 
provided  by  a  single  source.  IP-address-based  indicators  from  an  average  list  do 
not  provide  unique  value  to  CND  between  4.76%  and  17.54%  of  the  time.  This 
wider  range  for  IP-address-based  lists  is  expected  because  there  are  fewer  IP  ad¬ 
dresses  than  domain  names,  and  because  IP  addresses  are  more  commonly  reused. 
Therefore,  the  large  majority  of  the  time,  any  list’s  indicator  will  provide  unique 
information  and  value  to  CND. 

From  a  practical  point  of  view,  one  might  surmise  that  each  list,  or  perhaps 
a  pair  of  related  lists,  is  describing  and  following  a  specific  type  of  malicious 
behavior.  Each  of  these  malicious  behaviors  is  a  particular  kind  of  malicious 
behavior,  but  is  identified  differently  from  other  sorts  of  malicious  behavior.  We 
cannot  compare  one  list  to  another  list  to  determine  how  well  it  identifies  any 
particular  behavior,  because  each  list  is  idiosyncratically  following  a  different  kind 
of  behavior.  This,  in  turn,  means  that  there  is  no  thorough  or  convenient  way  to 
evaluate  the  performance  of  any  of  these  lists,  since  each  list  is  a  one-of-a-kind 
authority  on  the  particular  type  of  activity  it  detects. 
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A  further  difficulty  with  this  situation  is  that  there  is  no  ready  taxonomy  or 
terminology  for  describing  precisely  what  activity  a  malicious  actor  is  performing. 
Attempts  to  categorize  a  list  as  following  a  particular  malicious  activity  will  run 
into  terminology  and  communication  issues  between  researchers.  The  best  way  to 
determine  what  malicious  activity  a  list  is  following  is  to  know  what  algorithm  the 
list  uses;  however,  as  stated  above,  list  population  algorithms  are  understandably 
almost  never  shared.  This  leaves  both  the  academic  and  operational  cybersecurity 
community  with  few  resources  to  evaluate  efficacy. 

This  problem  is  especially  acute  for  academic  researchers  attempting  to  prove 
their  method  is  accurate  by  comparing  their  results  to  known  lists.  Most  lists 
do  not  intersect — and  if  they  do  intersect,  they  do  so  haphazardly — so  what  a 
researcher  considers  to  be  a  “good”  rate  of  intersection  to  prove  a  research  method 
accurate  may  be  meaningless.  Further,  it  is  important  to  consider  which  lists  are 
used  as  benchmarks,  since  so  few  common  public  lists  intersect. 

The  CND  take-away  from  this  analysis  is  that  any  one  list,  or  any  ten  lists, 
cannot  provide  a  comprehensive  description  of  all  malicious  indicators.  Every 
list  the  defender  can  obtain  and  use  will  probably  continue  to  provide  new,  non¬ 
overlapping  defense  to  the  network.  Though  the  defender  must  evaluate  the  qual¬ 
ity  of  new  identifiers,  any  new  list  can  provide  useful  identifiers  of  malicious  ac¬ 
tivity  not  already  contained  in  the  defender’s  list. 

A  CND  analyst  or  architect  can  also  conclude  that  blacklists  are  insufficient 
for  adequate  network  defense.  If  blocking  is  so  fragile,  it  is  too  easy  to  avoid. 
Other  established  methods  of  CND  should  be  prioritized  and  put  into  production 
as  appropriate,  such  as  gray  lists,  behavioral  analysis,  web  proxy  content  analysis, 
and  white  lists. 

These  blacklist  results  likewise  challenge  threat  intelligence  analysts.  Exist¬ 
ing  blacklists  should  be  used  to  examine  new  threats  with  caution.  Investigations 
certainly  cannot  rely  only  on  blacklists  for  the  detection  of  ongoing  activity.  Rep¬ 
utation  and  context  of  larger  units  of  the  Internet  become  increasingly  important 
to  get  a  better  idea  of  what  behavior  is  suspicious.  For  this  task,  processes,  such 
as  intelligent  indicator  expansion,  are  useful  [7], 
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Appendix:  Multi-page  Results 

Since  this  report  increases  the  total  number  of  lists  analyzed  from  25  to  85,  we 
cannot  support  the  same  level  of  detail  as  the  prior  report.  The  combinatorics  of 
the  pairwise  intersections  alone  run  many  tens  of  pages,  whereas  they  occupied 
only  a  few  pages  in  our  2013  report.  Therefore,  we  limit  the  rote  reporting  to  just 
the  pairwise  intersections  between  every  pair  of  two  lists  with  the  same  type  of 
indicator  (domains  or  IP  addresses).  Further  detail  would  more  likely  drown  the 
reader  in  detail  rather  than  provide  insights. 

Pairwise  Intersection  Counts 

Table  5:  Pairwise  intersections  for  lists  of  domains.  (Percentage  reported  is  of  the  smaller 
list  of  the  two.) 


Domain  Lists 

Intersect 

%  of  Smaller 

LD_1 

LD_2 

49 

0.01 

LD_1 

LD_3 

123 

0.22 

LD_1 

LD_4 

6044 

7.21 

LD_1 

LD_5 

13192 

17.98 

LD_1 

LD_6 

65 

0.14 

LD_1 

LD_7 

836 

1.25 

LD_1 

LD_8 

327 

9.35 

LD_1 

LD_9 

38802 

9.42 

LD_1 

LD_10 

27749 

11.05 

LD_1 

LD_11 

53168 

12.91 

LD_1 

LD_12 

32348 

7.85 

LD_1 

LD_13 

0 

0.00 

LD_1 

LD_14 

32973 

8.01 

LD_1 

LD_15 

7206 

11.73 

LD_1 

LD_16 

4 

0.16 

LD_1 

LD_17 

285 

0.16 

LD_1 

LD_18 

10691 

17.50 

LD_2 

LD_3 

266 

0.48 

LD_2 

LD_4 

228 

0.27 

LD_2 

LD_5 

283 

0.39 

LD_2 

LD_6 

9750 

20.40 

LD_2 

LD_7 

22 

0.03 

LD_2 

LD_8 

41 

1.17 

LD_2 

LD_9 

6257 

1.25 

LD_2 

LD_10 

1077 

0.43 

LD_2 

LD_11 

3096 

0.11 

LD_2 

LD_12 

1048 

0.07 

LD_2 

LD_13 

72 

41.62 

LD_2 

LD_14 

86669 

3.16 

LD_2 

LD_15 

140 

0.23 

LD_2 

LD_16 

264 

10.32 

LD_2 

LD_17 

15 

0.01 

LD_2 

LD_18 

249 

0.41 

LD_3 

LD_4 

90 

0.16 

LD_3 

LD_5 

379 

0.69 

LD_3 

LD_6 

119 

0.25 

LD_3 

LD_7 

8123 

14.74 

LD_3 

LD_8 

581 

16.61 

LD_3 

LD_9 

408 

0.74 

LD_3 

LD_10 

633 

1.15 

Domain  Lists 

Intersect 

%  of  Smaller 
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Domain  Lists 

Intersect 

%  of  Smaller 

LD_3 

LD_11 

21598 

39.19 

LD_3 

LD_12 

24300 

44.09 

LD_3 

LD_13 

0 

0.00 

LD_3 

LD_14 

1183 

2.15 

LD_3 

LD_15 

1334 

2.42 

LD_3 

LD_16 

9 

0.35 

LD_3 

LD_17 

518 

0.94 

LD_3 

LD_18 

651 

1.18 

LD_4 

LD_5 

9629 

13.13 

LD_4 

LD_6 

168 

0.35 

LD_4 

LD_7 

685 

1.02 

LD_4 

LD_8 

488 

13.95 

LD_4 

LD_9 

31829 

37.94 

LD_4 

LD_10 

31538 

37.60 

LD_4 

LD_11 

36496 

43.51 

LD_4 

LD_12 

22686 

27.04 

LD_4 

LD_13 

0 

0.00 

LD_4 

LD_14 

34926 

41.64 

LD_4 

LD_15 

5163 

8.41 

LD_4 

LD_16 

26 

1.02 

LD_4 

LD_17 

568 

0.68 

LD_4 

LD_18 

7963 

13.04 

LD_5 

LD_6 

199 

0.42 

LD_5 

LD_7 

1955 

2.92 

LD_5 

LD_8 

1218 

34.82 

LD_5 

LD_9 

59672 

81.35 

LD_5 

LD_10 

46846 

63.87 

LD_5 

LD_11 

71288 

97.19 

LD_5 

LD_12 

56766 

77.39 

LD_5 

LD_13 

0 

0.00 

LD_5 

LD_14 

51964 

70.84 

LD_5 

LD_15 

9123 

14.85 

LD_5 

LD_16 

17 

0.66 

LD_5 

LD_17 

513 

0.70 

LD_5 

LD_18 

20869 

34.16 

LD_6 

LD_7 

26 

0.05 

LD_6 

LD_8 

29 

0.83 

LD_6 

LD_9 

3394 

7.10 

LD_6 

LD_10 

700 

1.46 

LD_6 

LD_11 

894 

1.87 

LD_6 

LD_12 

566 

1.18 

LD_6 

LD_13 

51 

29.48 

LD_6 

LD_14 

21401 

44.78 

LD_6 

LD_15 

75 

0.16 

LD_6 

LD_16 

608 

23.76 

LD_6 

LD_17 

27 

0.06 

LD_6 

LD_18 

159 

0.33 

LD_7 

LD_8 

909 

25.99 

LD_7 

LD_9 

3400 

5.07 

LD_7 

LD_10 

3326 

4.96 

LD_7 

LD_11 

47226 

70.46 

LD_7 

LD_12 

52000 

77.58 

LD_7 

LD_13 

0 

0.00 

LD_7 

LD_14 

5030 

7.50 

LD_7 

LD_15 

2752 

4.48 

Domain  Lists 

Intersect 

%  of  Smaller 
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Domain  Lists 

Intersect 

%  of  Smaller 

LD_7 

LD_16 

6 

0.23 

LD_7 

LD_17 

677 

1.01 

LD_7 

LD_18 

1887 

3.09 

LD_8 

LD_9 

1257 

35.93 

LD_8 

LD_10 

1531 

43.77 

LD_8 

LD_11 

3459 

98.89 

LD_8 

LD_12 

2576 

73.64 

LD_8 

LD_13 

0 

0.00 

LD_8 

LD_14 

1404 

40.14 

LD_8 

LD_15 

360 

10.29 

LD_8 

LD_16 

3 

0.12 

LD_8 

LD_17 

213 

6.09 

LD_8 

LD_18 

1151 

32.90 

LD_9 

LD_10 

164295 

65.44 

LD_9 

LD_11 

248132 

49.69 

LD_9 

LD_12 

161632 

32.37 

LD_9 

LD_13 

11 

6.36 

LD_9 

LD_14 

343949 

68.88 

LD_9 

LD_15 

33077 

53.85 

LD_9 

LD_16 

67 

2.62 

LD_9 

LD_17 

754 

0.42 

LD_9 

LD_18 

45675 

74.77 

LD_10 

LD_11 

198030 

78.88 

LD_10 

LD_12 

126006 

50.19 

LD_10 

LD_13 

1 

0.58 

LD_10 

LD_14 

227320 

90.55 

LD_10 

LD_15 

21646 

35.24 

LD_10 

LD_16 

142 

5.55 

LD_10 

LD_17 

1363 

0.76 

LD_10 

LD_18 

34391 

56.30 

LD_1 1 

LD_12 

637635 

44.21 

LD_1 1 

LD_13 

2 

1.16 

LD_1 1 

LD_14 

245637 

8.97 

LD_1 1 

LD_15 

41458 

67.49 

LD_1 1 

LD_16 

184 

7.19 

LD_1 1 

LD_17 

5147 

2.88 

LD_1 1 

LD_18 

52263 

85.55 

LD_12 

LD_13 

1 

0.58 

LD_12 

LD_14 

172836 

11.98 

LD_12 

LD_15 

30508 

49.67 

LD_12 

LD_16 

89 

3.48 

LD_12 

LD_17 

2866 

1.60 

LD_12 

LD_18 

41572 

68.05 

LD_13 

LD_14 

38 

21.97 

LD_13 

LD_15 

0 

0.00 

LD_13 

LD_16 

0 

0.00 

LD_13 

LD_17 

0 

0.00 

LD_13 

LD_18 

0 

0.00 

LD_14 

LD_15 

27015 

43.98 

LD_14 

LD_16 

845 

33.02 

LD_14 

LD_17 

1318 

0.74 

LD_14 

LD_18 

38961 

63.78 

LD_15 

LD_16 

7 

0.27 

LD_15 

LD_17 

443 

0.72 

LD_15 

LD_18 

17879 

29.27 

Domain  Lists 

Intersect 

%  of  Smaller 
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Domain  Lists 

Intersect 

%  of  Smaller 

LD_16 

LD_17 

7 

0.27 

LD_16 

LD_18 

18 

0.70 

LD_17 

LD_18 

429 

0.70 

Domain  Lists 

Intersect 

%  of  Smaller 

Table  6:  Pairwise  intersections  for  lists  of  domains.  (Percentage  reported  is  of  the  smaller 
list  of  the  two.) 


IP-Address  Lists 

Intersect 

%  of  Smaller 

LI_1 

LI_2 

2432 

0.00 

LI_1 

LI_3 

10679 

0.29 

LI_1 

LI_4 

6 

0.03 

LI_1 

LI_5 

8493 

11.69 

LI_1 

LI_6 

7134 

32.06 

LI_1 

LI_7 

50 

0.00 

LI_1 

LI_8 

3 

0.01 

LI_1 

LI_9 

162 

0.03 

LI_1 

LI_10 

9 

0.04 

LI_1 

LI_1 1 

10 

0.02 

LI_1 

LI_12 

290 

0.01 

LI_1 

LI_13 

271 

0.05 

LI_1 

LI_14 

1 

0.00 

LI_1 

LI_15 

0 

0.00 

LI_1 

LI_16 

12 

0.05 

LI_1 

LI_17 

0 

0.00 

LI_1 

LI_18 

38 

0.06 

LI_1 

LI_19 

0 

0.00 

LI_1 

LI_20 

0 

0.00 

LI_1 

LI_21 

727 

0.01 

LI_1 

LI_22 

56 

0.03 

LI_1 

LI_23 

262 

0.02 

LI_1 

LI_24 

0 

0.00 

LI_1 

LI_25 

35 

0.03 

LI_1 

LI_26 

14 

0.04 

LI_1 

LI_27 

222 

0.02 

LI_1 

LI_28 

207 

0.03 

LI_1 

LI_29 

33 

0.06 

LI_1 

LI_30 

1 

0.00 

LI_1 

LI_31 

384 

0.02 

LI_1 

LI_32 

10 

0.04 

LI_1 

LI_33 

419 

0.01 

LI_1 

LI_34 

21 

0.05 

LI_1 

LI_35 

21 

0.06 

LI_1 

LI_36 

10 

0.04 

LI_1 

LI_37 

16 

0.07 

LI_1 

LI_38 

1114 

0.00 

LI_1 

LI_39 

9 

0.04 

LI_1 

LI_40 

107 

0.03 

LI_1 

LI_41 

115 

0.03 

LI_1 

LI_42 

524 

0.01 

LI_1 

LI_43 

132 

0.03 

LI_1 

LI_44 

44 

0.03 

LI_1 

LI_45 

0 

0.00 

LI_1 

LI_46 

153 

0.03 

LI_1 

LI_47 

96 

0.04 

IP-Address  Lists 

Intersect 
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IP-Address  Lists 

Intersect 

%  of  Smaller 

LI_1 

LI_48 

1142 

0.01 

LI_1 

LI_49 

847 

0.01 

LI_1 

LI_50 

732 

0.02 

LI_1 

LI_51 

100 

0.01 

LI_1 

LI_52 

28 

0.00 

LI_1 

LI_53 

652 

0.02 

LI_1 

LI_54 

4 

0.02 

LI_1 

LI_55 

0 

0.00 

LI_1 

LI_56 

25 

0.10 

LI_1 

LI_57 

848 

0.02 

LI_1 

LI_58 

0 

0.00 

LI_1 

LI_59 

419 

0.02 

LI_1 

LI_60 

128 

0.24 

LI_1 

LI_61 

0 

0.00 

LI_1 

LI_62 

23 

0.09 

LI_1 

LI_63 

61 

0.02 

LI_1 

LI_64 

6 

0.03 

LI_1 

LI_65 

7 

0.03 

LI_1 

LI_66 

38 

0.17 

LI_1 

LI_67 

2 

0.01 

LI_2 

LI_3 

172020 

0.27 

LI_2 

LI_4 

24 

0.00 

LI_2 

LI_5 

4195 

0.01 

LI_2 

LI_6 

570 

0.00 

LI_2 

LI_7 

38798 

0.06 

LI_2 

LI_8 

163 

0.00 

LI_2 

LI_9 

337147 

0.54 

LI_2 

LI_10 

1223 

0.00 

LI_2 

LI_1 1 

341 

0.00 

LI_2 

LI_12 

1262183 

2.01 

LI_2 

LI_13 

431535 

0.69 

LI_2 

LI_14 

2147 

0.00 

LI_2 

LI_15 

44 

0.00 

LI_2 

LI_16 

627 

0.00 

LI_2 

LI_17 

2 

0.00 

LI_2 

LI_18 

44538 

0.07 

LI_2 

LI_19 

0 

0.00 

LI_2 

LI_20 

13 

0.00 

LI_2 

LI_21 

4744289 

7.54 

LI_2 

LI_22 

139465 

0.22 

LI_2 

LI_23 

1002042 

1.59 

LI_2 

LI_24 

45 

0.00 

LI_2 

LI_25 

27055 

0.04 

LI_2 

LI_26 

10801 

0.02 

LI_2 

LI_27 

1261143 

2.01 

LI_2 

LI_28 

418279 

0.67 

LI_2 

LI_29 

38023 

0.06 

LI_2 

LI_30 

891 

0.00 

LI_2 

LI_31 

1488061 

2.37 

LI_2 

LI_32 

5016 

0.01 

LI_2 

LI_33 

3314532 

5.27 

LI_2 

LI_34 

22929 

0.04 

LI_2 

LI_35 

2484 

0.00 

LI_2 

LI_36 

785 

0.00 

LI_2 

LI_37 

10365 

0.02 

IP-Address  Lists 

Intersect 
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IP-Address  Lists 

Intersect 

%  of  Smaller 

LI_2 

LI_38 

4996421 

7.95 

LI_2 

LI_39 

459 

0.00 

LI_2 

LI_40 

326681 

0.52 

LI_2 

LI_41 

306717 

0.49 

LI_2 

LI_42 

2910147 

4.63 

LI_2 

LI_43 

400116 

0.64 

LI_2 

LI_44 

80052 

0.13 

LI_2 

LI_45 

73 

0.00 

LI_2 

LI_46 

318311 

0.51 

LI_2 

LI_47 

188167 

0.30 

LI_2 

LI_48 

9146038 

14.54 

LI_2 

LI_49 

6680645 

10.62 

LI_2 

LI_50 

3666567 

5.83 

LI_2 

LI_51 

625296 

0.99 

LI_2 

LI_52 

49680 

0.08 

LI_2 

LI_53 

3696190 

5.88 

LI_2 

LI_54 

1279 

0.00 

LI_2 

LI_55 

13417 

0.02 

LI_2 

LI_56 

16639 

0.03 

LI_2 

LI_57 

3254065 

5.17 

LI_2 

LI_58 

0 

0.00 

LI_2 

LI_59 

1083324 

1.72 

LI_2 

LI_60 

13267 

0.02 

LI_2 

LI_61 

605 

0.00 

LI_2 

LI_62 

12906 

0.02 

LI_2 

LI_63 

232722 

0.37 

LI_2 

LI_64 

886 

0.00 

LI_2 

LI_65 

2717 

0.00 

LI_2 

LI_66 

414 

0.00 

LI_2 

LI_67 

1502 

0.00 

LI_3 

LI_4 

569 

0.02 

LI_3 

LI_5 

42838 

1.15 

LI_3 

LI_6 

10917 

0.29 

LI_3 

LI_7 

6457 

0.03 

LI_3 

LI_8 

82 

0.00 

LI_3 

LI_9 

11358 

0.30 

LI_3 

LI_10 

656 

0.02 

LI_3 

LI_1 1 

1432 

0.04 

LI_3 

LI_12 

28060 

0.75 

LI_3 

LI_13 

14747 

0.39 

LI_3 

LI_14 

128 

0.00 

LI_3 

LI_15 

150 

0.00 

LI_3 

LI_16 

253 

0.01 

LI_3 

LI_17 

4 

0.00 

LI_3 

LI_18 

2244 

0.06 

LI_3 

LI_19 

10 

0.00 

LI_3 

LI_20 

41 

0.00 

LI_3 

LI_21 

50107 

0.87 

LI_3 

LI_22 

3726 

0.10 

LI_3 

LI_23 

12205 

0.33 

LI_3 

LI_24 

1 

0.00 

LI_3 

LI_25 

5075 

0.14 

LI_3 

LI_26 

3685 

0.10 

LI_3 

LI_27 

11774 

0.32 

LI_3 

LI_28 

10533 

0.28 

IP-Address  Lists 

Intersect 
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IP-Address  Lists 

Intersect 

%  of  Smaller 

LI_3 

LI_29 

3112 

0.08 

LI_3 

LI_30 

276 

0.01 

LI_3 

LI_31 

24464 

0.65 

LI_3 

LI_32 

1125 

0.03 

LI_3 

LI_33 

27032 

0.72 

LI_3 

LI_34 

1105 

0.03 

LI_3 

LI_35 

988 

0.03 

LI_3 

LI_36 

626 

0.02 

LI_3 

LI_37 

862 

0.02 

LI_3 

LI_38 

86978 

0.27 

LI_3 

LI_39 

272 

0.01 

LI_3 

LI_40 

3623 

0.10 

LI_3 

LI_41 

5021 

0.13 

LI_3 

LI_42 

18993 

0.51 

LI_3 

LI_43 

10557 

0.28 

LI_3 

LI_44 

1691 

0.05 

LI_3 

LI_45 

2 

0.00 

LI_3 

LI_46 

8996 

0.24 

LI_3 

LI_47 

5696 

0.15 

LI_3 

LI_48 

76497 

0.69 

LI_3 

LI_49 

52594 

0.55 

LI_3 

LI_50 

45608 

1.06 

LI_3 

LI_51 

4664 

0.12 

LI_3 

LI_52 

1840 

0.05 

LI_3 

LI_53 

24696 

0.59 

LI_3 

LI_54 

292 

0.01 

LI_3 

LI_55 

12 

0.00 

LI_3 

LI_56 

585 

0.02 

LI_3 

LI_57 

56236 

1.34 

LI_3 

LI_58 

1 

0.00 

LI_3 

LI_59 

46103 

1.23 

LI_3 

LI_60 

3069 

0.08 

LI_3 

LI_61 

125 

0.00 

LI_3 

LI_62 

1894 

0.05 

LI_3 

LI_63 

1832 

0.05 

LI_3 

LI_64 

226 

0.01 

LI_3 

LI_65 

521 

0.01 

LI_3 

LI_66 

282 

0.01 

LI_3 

LI_67 

246 

0.01 

LI_4 

LI_5 

12 

0.02 

LI_4 

LI_6 

10 

0.06 

LI_4 

LI_7 

2 

0.00 

LI_4 

LI_8 

3 

0.03 

LI_4 

LI_9 

2 

0.00 

LI_4 

LI_10 

0 

0.00 

LI_4 

LI_1 1 

0 

0.00 

LI_4 

LI_12 

4 

0.00 

LI_4 

LI_13 

10 

0.00 

LI_4 

LI_14 

0 

0.00 

LI_4 

LI_15 

0 

0.00 

LI_4 

LI_16 

0 

0.00 

LI_4 

LI_17 

0 

0.00 

LI_4 

LI_18 

1 

0.00 

LI_4 

LI_19 

0 

0.00 

LI_4 

LI_20 

0 

0.00 

IP-Address  Lists 

Intersect 
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LI_4 

LI_21 

7 

0.00 

LI_4 

LI_22 

2 

0.00 

LI_4 

LI_23 

2 

0.00 

LI_4 

LI_24 

0 

0.00 

LI_4 

LI_25 

0 

0.00 

LI_4 

LI_26 

1 

0.00 

LI_4 

LI_27 

5 

0.00 

LI_4 

LI_28 

3 

0.00 

LI_4 

LI_29 

0 

0.00 

LI_4 

LI_30 

0 

0.00 

LI_4 

LI_31 

6 

0.00 

LI_4 

LI_32 

0 

0.00 

LI_4 

LI_33 

5 

0.00 

LI_4 

LI_34 

0 

0.00 

LI_4 

LI_35 

21 

0.06 

LI_4 

LI_36 

2 

0.02 

LI_4 

LI_37 

1 

0.01 

LI_4 

LI_38 

7 

0.00 

LI_4 

LI_39 

0 

0.00 

LI_4 

LI_40 

0 

0.00 

LI_4 

LI_41 

5 

0.00 

LI_4 

LI_42 

19 

0.00 

LI_4 

LI_43 

1 

0.00 

LI_4 

LI_44 

0 

0.00 

LI_4 

LI_45 

0 

0.00 

LI_4 

LI_46 

6 

0.00 

LI_4 

LI_47 

2 

0.00 

LI_4 

LI_48 

16 

0.00 

LI_4 

LI_49 

11 

0.00 

LI_4 

LI_50 

13 

0.00 

LI_4 

LI_51 

2 

0.00 

LI_4 

LI_52 

1 

0.00 

LI_4 

LI_53 

6 

0.00 

LI_4 

LI_54 

0 

0.00 

LI_4 

LI_55 

0 

0.00 

LI_4 

LI_56 

0 

0.00 

LI_4 

LI_57 

14 

0.00 

LI_4 

LI_58 

0 

0.00 

LI_4 

LI_59 

11 

0.00 

LI_4 

LI_60 

5 

0.01 

LI_4 

LI_61 

0 

0.00 

LI_4 

LI_62 

0 

0.00 

LI_4 

LI_63 

1 

0.00 

LI_4 

LI_64 

0 

0.00 

LI_4 

LI_65 

1 

0.01 

LI_4 

LI_66 

1 

0.02 

LI_4 

LI_67 

0 

0.00 

LI_5 

LI_6 

9998 

13.76 

LI_5 

LI_7 

137 

0.00 

LI_5 

LI_8 

6 

0.01 

LI_5 

LI_9 

321 

0.05 

LI_5 

LI_10 

11 

0.02 

LI_5 

LI_1 1 

37 

0.05 

LI_5 

LI_12 

874 

0.03 

LI_5 

LI_13 

695 

0.13 

IP-Address  Lists 

Intersect 

%  of  Smaller 
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IP-Address  Lists 

Intersect 

%  of  Smaller 

LI_5 

LI_14 

2 

0.00 

LI_5 

LI_15 

2 

0.00 

LI_5 

LI_16 

26 

0.04 

LI_5 

LI_17 

1 

0.00 

LI_5 

LI_18 

141 

0.19 

LI_5 

LI_19 

0 

0.00 

LI_5 

LI_20 

0 

0.00 

LI_5 

LI_21 

1739 

0.03 

LI_5 

LI_22 

97 

0.06 

LI_5 

LI_23 

604 

0.05 

LI_5 

LI_24 

0 

0.00 

LI_5 

LI_25 

138 

0.12 

LI_5 

LI_26 

39 

0.05 

LI_5 

LI_27 

385 

0.03 

LI_5 

LI_28 

821 

0.14 

LI_5 

LI_29 

109 

0.15 

LI_5 

LI_30 

2 

0.00 

LI_5 

LI_31 

902 

0.05 

LI_5 

LI_32 

18 

0.02 

LI_5 

LI_33 

730 

0.02 

LI_5 

LI_34 

35 

0.05 

LI_5 

LI_35 

53 

0.07 

LI_5 

LI_36 

18 

0.02 

LI_5 

LI_37 

22 

0.03 

LI_5 

LI_38 

2285 

0.01 

LI_5 

LI_39 

13 

0.02 

LI_5 

LI_40 

168 

0.05 

LI_5 

LI_41 

208 

0.06 

LI_5 

LI_42 

1109 

0.03 

LI_5 

LI_43 

314 

0.06 

LI_5 

LI_44 

148 

0.09 

LI_5 

LI_45 

0 

0.00 

LI_5 

LI_46 

282 

0.06 

LI_5 

LI_47 

177 

0.07 

LI_5 

LI_48 

2466 

0.02 

LI_5 

LI_49 

2021 

0.02 

LI_5 

LI_50 

1615 

0.04 

LI_5 

LI_51 

243 

0.04 

LI_5 

LI_52 

32 

0.00 

LI_5 

LI_53 

1430 

0.03 

LI_5 

LI_54 

7 

0.01 

LI_5 

LI_55 

0 

0.00 

LI_5 

LI_56 

43 

0.06 

LI_5 

LI_57 

2026 

0.05 

LI_5 

LI_58 

0 

0.00 

LI_5 

LI_59 

978 

0.06 

LI_5 

LI_60 

103 

0.14 

LI_5 

LI_61 

5 

0.01 

LI_5 

LI_62 

55 

0.08 

LI_5 

LI_63 

63 

0.02 

LI_5 

LI_64 

11 

0.02 

LI_5 

LI_65 

12 

0.02 

LI_5 

LI_66 

14 

0.02 

LI_5 

LI_67 

11 

0.02 

LI_6 

LI_7 

41 

0.00 
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Intersect 
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LI_6 

LI_8 

4 

0.03 

LI_6 

LI_9 

93 

0.02 

LI_6 

LI_10 

10 

0.06 

LI_6 

LI_1 1 

12 

0.02 

LI_6 

LI_12 

138 

0.00 

LI_6 

LI_13 

135 

0.02 

LI_6 

LI_14 

1 

0.00 

LI_6 

LI_15 

1 

0.01 

LI_6 

LI_16 

11 

0.07 

LI_6 

LI_17 

0 

0.00 

LI_6 

LI_18 

15 

0.02 

LI_6 

LI_19 

0 

0.00 

LI_6 

LI_20 

0 

0.00 

LI_6 

LI_21 

361 

0.01 

LI_6 

LI_22 

25 

0.02 

LI_6 

LI_23 

103 

0.01 

LI_6 

LI_24 

0 

0.00 

LI_6 

LI_25 

29 

0.02 

LI_6 

LI_26 

10 

0.03 

LI_6 

LI_27 

104 

0.01 

LI_6 

LI_28 

94 

0.02 

LI_6 

LI_29 

15 

0.03 

LI_6 

LI_30 

1 

0.01 

LI_6 

LI_31 

214 
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